DOCUMENT RESUME 



ED 430 934 



SP 038 516 



AUTHOR 

TITLE 



SPONS AGENCY 

PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Rolheiser, Carol; Ross, John A.; Hogaboam-Gray, Anne 
Assessment in the Cooperative Classroom: Using an Action 
Research Enhanced Version of the Train the Trainer 
In-service Model To Impact Teacher Attitudes and Practices. 
Social Sciences and Humanities Research Council of Canada, 
Ottawa (Ontario) . 

1999-04-23 

26p.; Paper presented at the Annual Meeting of the American 
Educational Research Association (Montreal, Quebec, Canada, 
April 19-23, 1999) . Funding also provided by Bertelsmann 
Foundation. 

Reports - Descriptive (141) -- Speeches/Meeting Papers (150) 

MF01/PC02 Plus Postage. 

* Act ion Research; Cooperative Learning; Elementary Secondary 
Education; Evaluation Methods; * Faculty Development; Foreign 
Countries; *Inservice Teacher Education; Self Efficacy; Self 
Evaluation (Individuals) ; *Student Evaluation; Teacher 
Attitudes; *Teacher Improvement; Teachers 
*Train the Trainer 



ABSTRACT 



This research investigated the impact of combining two 
approaches to inservice teacher education (action research and train the 
trainer) on teacher attitudes and practices. The inservice developed 
assessment approaches aligned with cooperative learning instructional 
approaches. Teachers were introduced to a model of collaborative assessment 
aimed at improving students' abilities to self -evaluate . A team of five 
university and school/district personnel conducted a series of three 
inservice sessions over 4 months. Teams of teachers from several schools were 
paired, with two teachers attending the first full-day inservice as trainers, 
working back at the school with their partners (trainees) , then attending the 
second and third inservice sessions together. The effects of the inservice 
were measured in a pre-post design which examined the impact of the inservice 
design on personal teaching efficacy, general teaching efficacy, commitment 
to authentic practices, and commitment to traditional testing practices. Data 
were also collected on how teachers used the four stages in the study model 
for teaching self-evaluation. Researchers found significant pre-post changes. 
Use of self-evaluation procedures increased substantially, commitment to 
authentic assessment practices increased, commitment to traditional 
assessment practices declined, and personal teaching efficacy scores were 
higher. The action-research-enhanced train- the-trainer approach facilitated 
teachers' movement toward student assessment practices congruent with the 
ideals of the cooperative learning/teaching practices. This approach also 
succeeded in enhancing teachers' beliefs about their competence. (Contains 48 
references.) (SM) 
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The purpose of the research was to conduct an exploratory investigation of the impact of 
combining two approaches to in-service — action research and train the trainer approaches — 
on teacher attitudes and practices. This is a continuing study in a line of inquiry investigating 
cn assessment practices in the cooperative learning (CL) classroom. The content of the in-service 

o was on the development of assessment approaches aligned to cooperative learning (CL) 

^ instructional approaches. Specifically, teachers were introduced to a model of collaborative 
Q assessment aimed at improving students ’ abilities to self-evaluate. A team of five university and 
school/district personnel conducted a series of three in-service session over the course of four 
months. Participants from each school were paired, with two teachers attending the first full day 
in-service as trainers, working back at the school with their partners (trainees), and then 
attending the second and third in-service sessions together. The effects of the in-service 
conditions were measured in a pre-post design. The study examined the impact of the in-service 
design on personal teaching efficacy, general teaching efficacy, commitment to authentic 
.assessment practices, and commitment to traditional (psychometric) testing practices. In 
addition, data was collected regarding how teachers used the four stages in our model for 
teaching self-evaluation. The latter data were produced as a handbook of lesson plans, student 
products, and teacher reflections. We found significant pre-post changes: use of self-evaluation 
procedures increased substantially, commitment to authentic assessment practices increased 
while commitment to traditional assessment practices declined, and personal teaching efficacy 
scores were higher. The action research enhanced train the trainer approach was successful in 
facilitating teachers ’ movement toward student assessment practices congruent with the ideals of 
the cooperative learning teaching practices. This approach was also successful in enhancing 
teachers ’ beliefs about their competence. Although a great deal of experimentation in new forms 
of professional development has occurred, few studies have assessed the outcomes of different 
models. The findings of this study provide evidence of the relative effects of a combined 
approach to district-level in-service that capitalizes on the strengths of both a train the trainer 
approach and an action research approach. 
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Literature Review 



The Train the Trainer Approach 

In the train the trainer method, staff developers provide instruction to work group 
representatives rather than to all members of the unit. Recipients of direct training are expected 
to master the new material and then teach it to their peers in the home unit. The few studies of 
that have been conducted on this approach tend to report positive outcomes (e.g., Hester et al., 
1996; Wedman et al., 1996). 

The train the trainer method has several advantages. The first is economic: in-serving a 
few teachers may accomplish, at reduced cost, the same result as providing instruction for all 
teachers. Ross (1990), in a study of 64 grade 4 classes, found that t rainin g school representatives 
produced the same improvements in student problem solving as training all teachers. The results 
were relatively robust: the same finding was obtained regardless of whether the in-service was 
provided during or after school. Neef (1995), in the only other controlled study of the relative 
impact of the train the train approach, also found virtually identical student improvements for a 
train the trainer method (professionals trained parents who then trained other parents), as for a 
treatment in which a professional trained the whole sample directly. 

The second advantage of the train the trainer approach is cultural. Asking teachers to 
share teaching strategies with their peers increases opportunities for teacher leadership, 
communicates administrator confidence in the professional responsibility of staff, and reduces 
norms of isolation and privatism that impede the spread of innovations. Collaborative cultures 
are consistently associated with school improvement (Cousins, Ross, & Maynes, 1994; Hopkins, 
Ainscow, & West., 1994). 

The third advantage is that talk between teacher trainers and their trainees promotes 
teacher learning. In meta-analyses by Bennett (1987) and Yeany and Padilla (1986), PD designs 
that encouraged teachers to talk about the material to be learned produced more teacher change 
than other in-service activities. Professional sharing contributes to rehearsal of new ideas, 
identification of gaps in knowledge, and increases teacher reflection (Ross & Regan, 1993). 
Modeling of innovations by peers is likely to be more illuminating and motivating than modeling 
by experts (Bandura, 1997). 

The train the train approach also has disadvantages. It may create a status hierarchy, 
accentuating existing cleavages between those selected for the training and those at the end of 
the chain. Trained teachers may resist being cast in the role of expert and may prefer 
noninterventionist methods too subtle to overcome teacher resistance (Veslind & Jones, 1998). 
Teachers who have experienced a constructivist approach to teacher learning may communicate 
the products of their in-service experience but use a didactic approach in conflict with the 
process they experienced (Bencze, 1995; Bickel & Hattrup, 1995). The train the train approach 
may also founder due to the poor selection of trainers, lack of skill in the role, and too little time 
spent with the ultimate trainees. 
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The Action Research Approach 



The disadvantages of the train the trainer method might be reduced if an action research 
component is added. 

In order to identify the key attributes of action research it is useful to look at its origins. 
The principles underlying action research have a long history. The German philosopher G.W.F. 
Hegel (1771-1831) and educational theorist John Dewey (1860-1952) both “challenged the 
traditional idea that knowledge consists of eternal, universal principles discovered by 
intellectuals and passed on unchanged to the consumers of knowledge” (Beck, 1998, p49). Both 
viewed the first hand, concrete experiences of teachers in their classrooms to be essential in 
understanding educational principles and how they should be applied in specific contexts. The 
actual term “action research”, was first used in the 1940s by Kurt Kewin, and implied the 
purposeful “application of tools and methods of social science to immediate, practical problems, 
with the goals of contributing to theory and knowledge in the field of education and improving 
practice in schools” (Kemmis, 1980, as reported in Oja & Smulyan, 1989, p.l). Stephen Corey, 
an educational researcher in the 40s, was among the first to use action research in the field of 
education. He believed that changes in educational practice would be more likely to occur when 
teachers were involved in inquiry and the application of findings and he maintained that this 
form of inquiry was a valid source of original educational knowledge. He also emphasized the 
importance of collaboration in action research. Corey maintained “that if teachers do research 
together, the program modifications developed will be more feasible, there will be a greater 
commitment to change, a wider range of talents will be utilized and individual teachers will be 
less open to attack for their innovations” (Beck, 1998, p.50-5 1). While the 60s and early 70s saw 
little interest in action research in America, the time following saw renewed interest, especially 
because researchers were questioning the applicability of quantitative and experimental methods 
of research to educational issues and challenges. There was a need for methods that would 
capture complex classroom realities, and that would be useful decision-making tools for 
practitioners as they modified treatments throughout a process (Oja, & Smulyan, 1989). In the 
mid 70s new and expanded views of action research in education began to appear, including in 
the form of collaborative action research between teachers and university academics (e.g., 
Atwell, 1991). As McKeman notes, in this collaboration each “shares in planning, 
implementing, analyzing and reporting the research and... team members contribute unique skills 
and expertise in a collective process” (1998, p. 1 80). These new forms, including collaborative 
action research, were seen as simultaneously contributing to knowledge in the field, and 
improved practice in the classroom (Oja, & Smulyan, 1989). 

In examining this long history of action research there appears to be four basic elements 
that emerge. First is the stress on collaboration, second its focus on practical problems, third its 
emphasis on professional development, and fourth its need for a structure that allows participants 
time and support for dialogue (Oja & Smulyan, 1989). These attributes guided the design of our 
action research enhanced approach to the train the trainer method of in-service used in this study. 

In reviewing the history advantages and disadvantage also emerge. The advantages of 
action research projects coincide with three general aims of this approach: staff development, 
improved school practice and the modification and elaboration of theories of teaching and 



er|c 



4 



3 



learning. “Staff development through action research may take a number of forms, including 
increased teacher understanding of the classroom and school (Carr and Kemmis, 1986; Grundy 
and Kemmis, 1982; Nixon 1981); increased self-esteem resulting from active involvement in 
research, professional conferences, and perhaps publication (Elliott, 1985; McCutcheon, 1981; 
Sheard, 1981) and greater feelings of competence in solving problems and making decisions 
related to teaching and learning. Improved practice results from practitioner participation in the 
investigation of actions and issues of immediate importance.” (as reported in Oja & Smulyan, 
1989, pi -2). Action research helps teachers improve their individual practice in several ways. 
Teachers buffeted by reform movements launched from outside the school become more 
powerful when they have access to research data and tools (Schensul & Schensul, 1992). They 
may also be more willing to take professional risks if their involvement in research leads them to 
feel in greater control of their professional lives. Data obtained through one’s own efforts are 
more immediate and meaningful. Teachers are more likely to use research findings productively 
if the research provides a sense of ego involvement (Cousins & Earl, 1995; Cousins & Walker, 
1995). 

As well, teachers engaged in action research make important contributions to educational 
theory; theory that is importantly grounded in the realities of school. Action research encourages 
teachers to become instructional theorists. When teachers conduct research they articulate their 
intentions, test assumptions, and make connections among elements of their practice (Cochran- 
Smith & Lytle, 1990). Their contributions often include the discovery and elaboration of 
theoretical frameworks underlying their practice (Carr and Kemmis, 1986). 

► 

Although not every action research project aims at or meets all of these goals — staff 
development, improved school practice and the modification and elaboration of theories of 
teaching and learning - most include elements of all three (Oja & Smulyan, 1989). Some 
educators believe that these combined advantages might also lead to enhanced teacher status and 
morale, as teachers’ research activities are shared with the broader educational community 
(Beck, 1998). 

Case studies of action research have supported the expectations and advantages listed 
above. Teachers who engaged in action research indicated that they became more reflective 
about their instructional practices during the inquiry (Caro-Bruce & McCreadie, 1994). Some 
teachers reported that their action research project convinced them to change their teaching, for 
example, by giving more attention to prerequisite knowledge and skill (Buckmaster, 1994), 
introducing strategies for promoting gender equity (Stroeher, 1994), integrating project-based 
activities into a traditional science curriculum (Scott, 1994) or shifting from traditional to 
authentic student assessment practices (Ross, Rolheiser, & Hogaboam-Gray, 1999). Others 
reported that action research gave them greater insight into students’ thinking, for example, 
concerning students’ cognitions about their teacher’s assessment practice (Stuart, 1994). Still 
others have credited action research with motivating teachers to change the context in which 
teaching occurs, for example, by extending teacher collaboration within the school (Simms, 

1994) or motivating teachers to challenge assessment policies that fail to acknowledge the out- 
of-school achievements of minority students (Kester, 1994). 
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Almost all studies of action research examine effects on teachers or students in isolation, 
without comparison to other in-service treatments. The only reported study to compare action 
research to another professional development found significant student benefits for the former 
(Ross, Rolheiser, & Hogaboam-Gray, 1998). In this study 23 classes were randomly assigned to 
two frequently used professional development methods. In the skills training treatment expert 
presenters provided strategies for teaching students how to evaluate their work. Teachers were 
expected to implement the strategies presented. In the action research treatment teachers 
received the same resources but the in-service sessions emphasized the process used by a 
previous group of teachers to improve their use of student self-evaluation. Pre- and post-test 
student surveys and interviews indicated that the action research condition made a more positive 
contribution to student attitudes toward self-evaluation than the skills training condition. The 
advantage of the action research condition was attributed to (1) sharing control in the in-service 
provided a better model of sharing control in the classroom, and (2) the handbook examples 
provided to teachers in both conditions were generated in earlier action research projects, making 
the information more accessible to teachers in the action research condition. 

There are also disadvantages of action research. There is concern that action research is 
still not strongly embraced and legitimate in education because it is undermined by current 
paradigms of educational research and practice (Beck, 1 998). Many researchers still do not 
value it as a recognized form of knowledge production. Those who do not value action research 
do not see the importance of drawing on teachers’ experience to identify questions neglected by 
researchers, nor see how the interpretation of findings might be enriched with the tacit 
knowledge of teachers (Lytle & Cochran-Smith, 1990). Likewise, .in contexts where teacher 
decision-making, initiative, and experimentation are not valued, it is difficult to find the support 
for action research. Accordingly, many preservice and inservice programs and activities are 
“typically organized to disseminate a knowledge base constructed almost exclusively by outside 
experts” (Cochrane-Smith & Lytle, 1993). Another disadvantage of action research is that many 
teachers lack skill in research methods. This is often a problem affecting even teachers with 
formal training in conducting research (Green & Kvidahl, 1990). Another disadvantage or 
challenge associated with action research is that teacher involvement is also limited by lack of 
time to do research. In addition, cultural norms giving primacy to classroom instruction over all 
other teacher roles constrain teachers participation in research. If workplace conditions are not 
conducive, action research can end up being an add-on activity for teachers, and consequently 
difficult to maintain as a regular practice. 

The combined in-service strategy used — an action-research enhanced approach to the 
train the trainer method — was utilized in a belief that the combined advantages of the two 
methods would outweigh the disadvantages of using either approach as the sole strategy. The 
attributes of the combined strategy included: 1) staff developers providing instruction to work 
group representatives; 2) the recipients of direct training were expected to master the new 
material and then teach it to their peers in the home unit using a collaborative model; 3) initial 
focus of the action research and follow up support were intended to focus on practical problems; 
4) an emphasis on professional development was maintained; and 5) an overall structure that 
provided participants time and support for dialogue. 

Based on the literature review we anticipated that the combined approach would 
contribute to: 1) the improved practice of individual teachers; 2) teachers’ sense of control of 
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their work lives; 3) teachers becoming more reflective about their practice, becoming 
instructional theorists; 4) the development of a collaborative culture of inquiry; and 5) teachers 
challenging the system politically. 



Table 1 About Here 



Research Question 

We conducted an exploratory investigation of the effects of a combined in-service 
strategy. Our inquiry was guided by the general question, “will an action research enhanced 
approach to the train the trainer method of in-service contribute to teacher learning?” We used 
quantitative and qualitative techniques to address the question. 

Method 



Sample 



A joint school-university team that had been developing student assessment strategies for 
cooperative learning classrooms over a five-year period initiated the project. Schools in one 
region of a large school district were invited to identify a team of two teacher leaders (the 
trainers) and two other teachers (the trainees) who were using cooperative learning regularly and 
were interested in working on their student assessment strategies. Twelve of 14 schools chose to 
participate. They identified 43 teachers 2 : 61% were from elementary schools and the remainder 
from secondary. The sample was 78% female and relatively young (mean experienced 2. 8 
years). The teachers had participated in a variety of professional programs (58% had additional 
university qualifications) but only one had a graduate degree. 

Sources of Quantitative Data 

At the beginning and end of the study, all teachers completed a survey containing five 
scales. Use of Self-Evaluation consisted of 12 Likert items measuring teachers’ self-reported use 
of self-evaluation strategies. For example, “my students and I brainstorm together the criteria for 
their self-evaluation.” The items were generated from the four stages of teaching student self- 
evaluation (to be described below) for this study. Half the items were negatively worded. 

Support for Authentic Student Assessment” measured teachers’ commitment to authentic 
assessment practices (e.g., performance tasks, portfolios, self-evaluation) and the extent to which 
teachers support the values of authentic assessment (e.g., student participation in evaluation 
decision making). For example, “I use parent-teacher conferences to communicate student 
learning”. Support for Traditional Testing was a parallel scale measuring teachers’ commitment 
to traditional test practices (e.g., tests should be administered to individual students at the end of 
units) and values associated with these practices (e.g., secrecy). For example, “My primary 
purpose for assessment is to assign grades to students.” The 26 Likert items in these two scales 
were selected from 52 items developed by Haydel, Oescher, Kirby, & Brooks (1997). The factor 
loadings reported by Haydel et al. were used to select the items. ' 



2 Five teams had only 3 members. 
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There were 16 Likert items measuring teachers’ expectations about their professional 
abilities. The items were adapted from Gibson & Dembo (1984), the most frequently used 
teacher efficacy instrument. Personal Teaching Efficacy measured teachers’ confidence in their 
teaching abilities, focusing on their ability to teach students how to evaluate their work. For 
example, “when a student improves at self-evaluation, it is usually because I found more 
effective approaches to teaching self-evaluation.” Gibson and Dembo’s original instrument 
referred to general expectations about teacher ability to bring about student learning. The items 
were modified to focus on teaching self-evaluation because social cognition theory (Bandura, 
1997) suggests that self-efficacy varies within-individuals and there is growing evidence of 
within-teacher variability on teacher efficacy measures (Raudenbush, Rowan, & Cheong, 1992; 
Ross, Cousins, & Gadalla, 1996). General Teaching Efficacy measures teachers’ beliefs about 
the ability of the teaching profession to bring about learning in all children. For example, “The 
amount that a student can learn is primarily related to family background.” Teachers with high 
scores on either of these teacher efficacy scales are more likely than teachers with low scores to 
set high expectations for their students and themselves, persist through obstacles, and have 
higher student achievement (Ross, 1998). 

Sources of Qualitative Data 

The major source of qualitative data were reports of the action research projects of the 
teachers (Laginski & Llyod, 1998). For example, one teacher had grade 3 mathematics students 
assess their pet home construction projects. Teacher and students developed a rubric together; 
students used the rubric to self- and peer-evaluate; the teacher conferenced with each student to 
triangulate the self-, peer-, and teacher-evaluations of the child’s pet home and to set learning 
goals for mathematics. Each teacher recorded his or her project on a planning template, with 
attachments, that was designed to communicate the products of the in-service in sufficient detail 
to enable another teacher to use the ideas in his or her own classroom. This template provided the 
agenda for teachers to share the results of their work with other teachers attending the final in- 
service. The three investigators compiled field notes on teacher interactions at the sessions. 
Teachers also completed an oral and written self-evaluation (“What did you learn? How did your 
kids benefit? What advice would you give to others/using adapting your ideas?”) and a peer 
evaluation. In the peer evaluation teachers received oral advice from one school partner (“One 
idea I really liked is. . .because. . . One question I have is. . .”) which the recipient of the peer 
evaluation recorded. 

Treatment 



The five staff developers responsible for designing and delivering the treatment were a 
combination of teachers and university personnel who had been engaged in inquiry and staff 
development related to the use of student self-evaluation strategies over the five previous years. 
Together with other members of the CLEAR group (a school-university collaborative inquiry 
partnership), these five staff developers had been engaged in devising and refining a four-stage 
model for teaching students how to evaluate their work: (i) involve students in setting the criteria 
on which they will be evaluated; (ii) model the criteria; (iii) give feedback on student 
understanding of the criteria; and, (iv) help students use self-evaluation data to set goals 
(Rolheiser, 1996). The in-service procedures were a refinement of our previous efforts to 
develop an action research approach to professional development (Ross et al, 1998). 
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In the first session, the two teacher leaders (trainers) from each school attended an initial 
full day session in January (9:00-4:00) to learn strategies for teacher self-evaluation, and 
strategies to work with their two partners (trainees) following the first session. The design of the 
initial day included a self-assessment of each participant’s current use of student self-evaluation, 
and hearing an overview of the project, including goals, timelines, and participant expectations. 
The researchers on the staff development team provided an overview of the theory and research 
behind student self-evaluation, and provided some guidelines for helping students understand 
what self-evaluation is and why it is important. Next, the trainers participated in an experiential 
activity designed to introduce them to the entire four-stage model for self-evaluation, using 
narrative writing (the introduction to a horror story) as the focus. After the experience the 
participants carried out a critical analysis of the “Woes and Wows” of each of the four stages. 
This activity allowed them to predict problems they might have in implementing each stage in 
their classrooms, as well as anticipating advantages of working through such challenges. Each of 
the trainers received a copy of a self-evaluation handbook (Rolheiser, 1 996) with case studies of 
the processes and products of action research, along with practical instruments and forms to 
support each of the four stages of the self-evaluation model. The participants had an opportunity 
to examine and use the book throughout the session. The session also included a brainstorming 
activity designed to generate ways that the trainers could engage their partners back at the 
schools in the learning that the trainers had experienced during the first in-service. The session 
ended with surveys distributed for all participants. 

The action research enhanced train the trainer treatment encouraged all participants to 
replicate the four-stage model, but it was up to them to determine how they would construct this 
in their different settings. To support such experimentation, the expectation was communicated 
in the first session that both trainer and trainee would attend the second in-service being prepared 
to report what actions they had jointly carried out around student self-evaluation. To this end, 
the trainers were provided an “In-school Meeting” structure to guide their meetings with their 
trainees between session one and two (see Appendix 1). The format included an interview and 
written recording for all participants regarding what they did the past week, successes 
experienced, and problems or concerns experienced. A process for problem-solving “Barrier- 
Busting” (see Appendix 2)was modeled for trainers in the first session, and included as a process 
for the in-school meetings. Trainers were asked to bring their in-school meeting recording sheets 
to the second in-service. To support the in-school collaboration all trainees received two full 
days of in-school release time to conference with and train their school-based partners over the 
course of the treatment. The school district also asked all teachers involved in the project to 
contribute after hours time equal to two days. 

In the second session, approximately four weeks later, both trainers and trainees attended 
an after school session together (4:00-7:00 p.m.). School teams began the session by sharing 
what they did since the first session, with a focus on successes. The staff developers modeled a 
visual flow chart method to record the participants’ initial steps. The process involved trainers 
interviewing their trainees (and then vice versa), and doing a visual flow chart that recorded the 
key events of each person’s experimentation with self-evaluation (see Appendix 3). The final 
bubble on the flow chart was a response to: “ What did you want: to do that you didn’t do this 
time?”. When finished the interview the partners labeled the flow chart with the stages of the 
four-stage model they experimented with. The interview highlights were shared in the large 
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group and encouragement provided to experiment with any stages not tried. Additional ideas 
were provided by the staff developers regarding the four-stage self-evaluation model. Next, one 
of the staff developers modeled the “Barrier-Busting” process once again. Finally, the 
participants were set up for a culumination activity to be shared at the last in-service session nine 
weeks later. All participants were provided with a lesson template for recording a key lesson 
targeted to some or all stages of the four-stage model, to be brought to the final in-service, along 
with sample artifacts from that lesson. 

In the final half-day session two months later, trainers once again attended during the day 
with their trainees. Goals for the session were shared, which included the celebration of 
successes, the identification of good ideas to share with other teachers, focussed reflection, and 
planning next steps for continued growth. The session started with a community circle, where 
each teacher shared one word that reflected what they or their students had learned from self- 
evaluation. Then, groups of three were formed across schools (elementary schools together, and 
secondary schools together). One person shared a learning experience they organized for their 
students as part of their action research, while the other two provided positive feedback (“What I 
liked...”) and raised questions for clarification or to probe. This process of sharing teacher plans, 
student artifacts, and teacher reflection was done orally and in writing. All three participants 
carried out this process. Upon completion, each person was asked to complete an individual 
reflection sheet with three questions (“As a teacher, what did you leam?; How did your students 
benefit?; and, What advice would you give to others using or adapting your ideas?”). The final 
part of the third in-service involved elementary and secondary pairs joining together. Each 
participant once again independently completed the self-assessment profile completed in the first 
session. Each person compared their growth with the previous profile and discussed this with 
their colleagues, providing evidence of growth where possible. Finally, surveys were completed 
and lesson templates/artifacts submitted. 

Analysis 

Descriptive statistics (means, standard deviations, internal consistencies) were compiled 
for the quantitative data. Pre to post differences were assessed using t-tests and effect sizes were 
calculated using Glass, McGaw & Smith’s (1981) procedures. Themes were developed from the 
qualitative data using analytic induction (scanning the data for categories and relationships 
among them), constant comparison (checking responses against other data for the same case), 
and triangulation (of data sources and judges) (Creswell, 1998). 

Results 



Quantitative Results 

Table 2 displays the means, standard deviations, and instrument reliabilities before and ' 
after the in-service. The internal consistencies of the scales were adequate (most were in the.70s 
and .80s). Table 2 shows that scores changed in expected ways. After the in-service teachers’ 
self-reported use of ‘self-evaluation was up substantially (ES=1.43, a large effect). There were 
small but significant changes in teachers’ beliefs about student assessment. Teacher support for 
authentic assessment increased (ES=34) and support for traditional testing declined (ES—.23). 
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Personal teaching efficacy (teacher confidence in their ability to use self-evaluation) significantly 
improved (ES=.60. a moderately large effect). General teaching efficacy, which was not directly 
targeted in the in-service, did not change significantly. These quantitative data suggest that the 
in-service was effective in meeting its goals. 

Table 2 About Here 



Qualitative Results 

Theme 1 Teachers enacted the practices recommended in the in-service. 

All teachers included artifacts (lesson plans, instruments, examples of student work) that 
they had developed for their own grades, subjects and topics in response to the in-service. All 
four stages of teaching self-evaluation outlined in the in-service were covered in every school but 
not by every teacher. Teachers were twice as likely to develop activities around Step 1 (involving 
students in setting the criteria for the evaluation) than the other steps: 84% provided information 
indicating that students were involved in developing a checklist of skills, a set of criteria with the 
highest level described or, most frequently, a rubric in which 2-4 levels of performance were 
described for 3-5 criteria. The remaining teachers also provided checklists and rubrics but it was 
not possible to tell from the information given what role students played in developing 
assessment criteria. Each of the remaining stages were overtly addressed by half as many 
teachers as attempted Stage 1 . The most common practice was to enact Stage 1 followed by two 
or more of the remaining stages. 

Exhibit 1 provides a description of a typical teacher project (using self-evaluation to 
strengthen paragraph writing of grade 9 students). Exhibit 2 displays one of the rubrics the same 
teacher developed for her class (S.E.E.C. in the second row is a mnemonic familiar to students in 
the class). This example [39a 3 ] illustrates key ideas presented in the in-service: development and 
refinement of a rubric with extensive student involvement; creation/selection of anchor papers to 
instantiate the rubric descriptions; triangulation of self-, peer-, and teacher-evaluations; and goal 
setting. 



There were few elementary-secondary differences. A higher proportion of secondary 
teachers explicitly required students to set goals and actions based on their self-evaluations. 
Although elementary teachers were less likely to include formal goal setting, the elementary 
teachers who did were more likely than secondary teachers to schedule a teacher-student 
conference. 

All teachers indicated that they had learned something about teaching from using self- 
evaluation in their classrooms. What they learned was highly individual— 22 different categories 
of response were given. For example, several teachers commented that consensus could be 
reached around expectations, including a grade 2 teacher who stated “I learned that students have 
similar ideas to me in what is expected of them. They just need help in wording the 



3 The references to the qualitative data use the form: teacher ID#, followed by the data section (a=artifacts, s=student 
benefits, t=teacher statements about what he or she learned, ad=advice teachers offered to those not in the in-service, 
p=peer feedback from a teacher in the in-service, q=question posed by a peer) 
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expectations.” [19t] Other teachers noted the importance of precision in stating expectations, 
including a grade 6 social studies teacher who indicated that the “wording of criteria is very 
important and needs to be most specific.” [15t] Some teachers, particularly those with younger 
students, were pleasantly surprised about the ability of students to evaluate their work. A grade 
5-6 science teacher reported that “students can set reasonable criteria for themselves with very 
little guidance. . .students (majority) are able to evaluate their performance honestly and set 
reasonable goals for [the] next task.” [20t] Other teachers reported a loosening of constraints 
around evaluation procedures. For example, a grade 1 1 creative writing teacher wrote “It’s okay 
to take some of the onus off the teacher”. [3 It] 

When asked what advice they would give to teachers using the ideas and materials they 
produced in the project, the responses were again highly individual with 33 different categories 
offered. Most of the suggestions were confirmations of experience congruent with the 
expectations of the in-service. For example, from a grade 2 teacher, “the criteria must be 
developed with the students in their own language” [5ad], from a grade 10 Family Studies 
teacher, “collect student work to show as examples” [4 lad], and from a grade 12 Art teacher, “be 
willing to adapt or compromise in order to progress.” [38ad] Some of the teachers described 
changes that they would make in their procedures or their instruments. For example, a grade 9 
Art teacher suggested that teachers using her lesson plans should “be willing to go beyond 
content into learning strategies.” [38ad] Other suggestions focused on things that teachers had 
not done but thought they should do. For example, teachers recommended that self-evaluation be 
done throughout the year so that students could see their growth over time, that a peer evaluation 
component be added, that goal setting be formalized, and that teacher-student conferences to 
interpret evaluation data be arranged. These suggestions were all congruent with the goals of the 
in-service. 

Theme 2 Teachers developed personal rationales for including self-evaluation in their teaching. 

All teachers articulated a personally meaningful rationale for using self-evaluation in 
their classrooms, based on benefits for students. Although 17 different types of benefits for 
students were identified, six were particularly common. In order of frequency: (i) Self-evaluation 
increased student clarity about expectations for particular assignments. For example, a grade 7 
English teacher reported that “students knew exactly what was expected and how it would be 
marked” [16s]. (ii) Teachers believed that students developed a sense of ownership and a greater 
sense of control of the evaluation process. This example, from a grade 12 art teacher, was 
typical: “The students learned that their ideas are valid and that they have a direct input into their 
grade.” [38s] (iii) Many teachers believed that the quality of student performance had improved 
as a direct result of self-evaluation training. A grade 3 mathematics teacher reported that: 
“[Students] produced better products because the children designed and used the criteria. When 
discussing completed work, both the teacher and the student work from the same level.” [Is], (iv) 
Teachers believed that self-evaluation engendered a stronger sense of student responsibility for 
their work. For example, from a grade 8 language teacher, “students take on greater 
accountability when they are involved in setting up the criteria.” [3s] (v) Teachers argued that the 
clarity of expectations provided by self-evaluation increased student motivation. For example, a 
grade 6 social studies teacher stated: “Students will rise to meet the higher performance levels. 
[They] strive for success when it’s been determined what they have to do to achieve.” [9s] Some 
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teachers indicated that self-evaluation’s contribution to motivation affected only a portion of 
their class. From a grade 1-2 mathematics teacher, “[self-evaluation] made many choose to bring 
their work up to a higher level.” [23s] (vi) Teachers believed that self-evaluation increased 
student self-knowledge, as indicated by a grade 4 social studies teacher “They knew their areas 
of weakness and could improve on them when given another assignment.” [7s] 

Theme 3 Teachers asserted rather than demonstrated their claims about the contribution of self- 
evaluation to student and teacher learning. 

Teachers made a broad array of claims about the benefits of self-evaluation but they did 
not provide any evidence to support these claims, nor were they asked to do so by their peers. 
When we asked teachers how they knew, for example, that student performance had increased, 
the data sources were ambiguous, the standards of comparison unclear, and procedures to ensure 
the credibility of the findings were virtually absent. 

Theme 4 Teachers learned from each other. 

Each teacher produced an individual project. Although most teachers identified a unique 
area of the curriculum to work in, there were some commonalities within schools. One 
elementary team focused on science with the kindergarten, grade 2, 4, and 8 teachers each 
focusing on using self-evaluation on a performance task appropriate to the grade. One high 
school team focused on different skills in the visual arts. There was also a pair of teachers in one 
school who worked together on language and three teachers in another school who worked on 
oral presentation skills. 

Collaborative learning was most visible in the peer evaluation process. Teachers were 
asked to indicate one aspect they liked and to pose one question in response to a peer’s project. 
The items that teachers liked in the projects of their peers were the same features, described 
above, that they liked in their own work. For example, 

I liked the way that you really gave your students input into their own learning. Your 
rubrics are very detailed and organized and it sets [students] up to be accountable. The 
students know exactly what is expected of them and I’m positive that they learned a lot 
by both creating the criteria for the rubric and applying it to their performance. [12p] 

In most instances the questions posed by peers asked how the teacher was going to follow 
up the self-evaluation activities with the same activities in different topics or subjects. There 
were also questions that implied criticisms or made tacit suggestions. For example, a grade 2-3 
teacher who gave students a teacher constructed rubric for evaluating their use of the writing 
process was asked “How would you go about working with the students to establish their own 
criteria? Would you focus on one aspect of the process or set up a rubric for each stage?” [14q] 

In her reflections the teacher offered advice to teachers (who might use the materials she 
developed in the project) that incorporated the peer feedback she received “Students would be 
involved in setting criteria and designing a rubric for each stage of the writing process.” [Mad] 
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We identified 13 instances in which the peer questions implied changes the teacher could 
make. The recipients of the feedback, as a group, covered fewer of the stages in teaching self- 
evaluation than the rest of the sample. The feedback encouraged them to attend to a stage they 
had not addressed in their project. In ten of these instances teachers responded with advice for 
others that incorporated the suggestions. For example, a grade 7-8 English teacher who focused 
exclusively on involving students in defining the highest level of performance on a checklist for 
assessing oral presentations was asked “Did they over or under estimate themselves? Perhaps a 
space left for comment for justifying the numerical rating they give themselves...” [27q] The 
teacher’s advice to others was “give room for kids to justify mark on evaluation.” [27ad] 

Summary of Results 

The quantitative and qualitative data present a similar picture. The in-service contributed 
to teacher learning of student evaluation skills. Teachers’ self-reported use of self-evaluation 
increased and the artifacts demonstrated that they had enacted in contextually appropriate ways 
the procedures recommended in the in-service. Teacher support for the values of authentic 
assessment strengthened and their commitment to traditional testing declined, both overt goals of 
the in-service. The qualitative data indicated that teachers had identified personal rationales for 
using self-evaluation in their classrooms, mainly based on perceived student benefits, and they 
espoused the same beliefs in offering advice to their peers. There was also evidence that teachers 
learned from each other, offering peer feedback congruent with the goals of the in-service. 
Finally, teachers’ confidence in their ability to use self-evaluation increased significantly during 
the study, while general expectations about the efficacy of teaching, an area not overtly 
addressed in the in-service were unaffected. 

Discussion 

The quantitative and qualitative results pointed in the same direction. The action research 
enhanced version of the train the trainer model had a positive impact on teacher learning. The 
effect sizes, for the quantitative data, were statistically significant and ranged from small to 
large. 



Since this exploratory study did not have a control group, some comparisons are required. 
In general, the results of in-service reported in previous research have been disappointing due to 
•the predominance of brief one-shot sessions unrelated to teachers’ perceived needs. Better results 
have been reported for approaches that share many of the characteristics of the in-service model 
used in this study, such as teacher control of implementation, classroom experimentation, 
reporting successes and failures to peers, and setting new goals for future professional growth 
(Stallings, 1989 inFullan, 1991). 

Some of the measures used in this study have been included in other studies. Teacher 
efficacy tends to be a stable trait, after the initial years of teaching, which is not easily influenced 
through planned interventions. A review of 1 1 studies that attempted to heighten teacher 
expectations of their teaching abilities through skills-based in-service found mixed results (Ross, 
1998). Even the few successful projects produced small effect sizes. The review found that 
teacher expectations were more likely to improve when teachers implemented the prescriptions 
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of the in-service and discussed interpretations of their experiences with peers, two processes 
included in the model in this study. 

The other measures of teacher growth used in this study were also included in an 
unreported study of the effects of graduate instruction on teachers’ assessment beliefs and 
practices. There was a significant increase in self-reported use of student evaluation during the 
graduate course. The effect size (.80) was lower than in the current study and there were no 
significant differences on teacher support for authentic or traditional testing cultures. These 
comparisons suggest that the in-service model used in this study contributed to teacher learning, 
although without a control group threats to the internal validity of the findings cannot be 
discounted. 

Although adding an action research component to the train the trainer model strengthened 
the in-service, the full benefits of the action research approach were not realized in this study. 
There was little evidence of a culture of inquiry developing. The teachers did not collect 
systematic data on the effects of their changed practice and did not look for evidence when 
reflecting on the projects of others. The most likely reason is that no specific training in research 
skills was provided, no models of appropriate data were displayed, and no importance was 
attached by the presenters to the credibility of claims. These are remediable problems. We regard 
the current exploration as a promising start to build on. In our subsequent efforts we plan to 
integrate design and measurement issues into the agenda. 
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Table 1 Advantages and Disadvantages of Two In-service Methods 



In-service Method 


Advantages 


Disadvantages 


Train the Trainer 


.low cost 


.creates status hierarchy 




.improved school culture 


.ineffective change strategies 




.teacher learning 


.insufficient time 


Action Research 


.improved practice 


.lack of research skill 




.increased teacher control 
.greater teacher reflection 
.culture of inquiry developed 
.challenge to system 


.insufficient time and resources 



Table 2 Means, Standard Deviations, Reliabilities and T-tests of Outcome Variables 



Pretest (n=43) Posttest (n=43) 



Variables 


Mean 


SD 


alpha 


Mean 


SD 


Alpha 


T 


Use of Self-Evaluation 


3.73 


.82 


.82 


4.90 


.57 


.85 


-9.66*** 


Authentic Assessment 


3.48 


.46 


.76 


3.74 


.41 


.77 


-4.77*** 


Traditional Testing 


2.59 


.52 


.85 


2.47 


.37 


.78 


2.27* 


Personal Teaching Efficacy 


3.97 


..53 


.67 


4.29 


.65 


.82 


-4.01*** 


General Teaching Efficacy 


3.72 


.74 


.73 


3.85 


.61 


.60 


-1.39 



***£<.001 **£<.01 *£<.05 
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Exhibit 1 



Example of Procedure for Teaching Self-Evaluation in Grade 9 English 

Directions: 

1. The teacher introduces 'self-evaluation'. What is it? What are the pros and cons? 

2. Groups brainstorm elements of descriptive paragraphs, and then participate in a whole 
class discussion about which criteria is particularly important for effective writing of a 
descriptive paragraph. 

3. Groups choose 3 criteria; discussion follows. 

4. Class chooses 3 criteria to base rubric. 

5. Rough draft of.rubric prepared by teacher then brought back for revisions. 

6. Final draft of rubric prepared; each student receives a copy. 

7. Groups are asked to write a descriptive paragraph for each level: low-middle-high. 

8. After each group has submitted their 3 examples of descriptive paragraphs, groups 
receive copies of all paragraphs. 

9. Class decides which "group" of 3 paragraphs is the best example of low-middle-high 
levels. 

10. Each student now receives a copy of this final draft of examples. 

1 1 . Opportunity for each student to write a descriptive paragraph. 

12. Students self-evaluate their descriptive paragraphs using rubric. 

13. Descriptive paragraphs are now peer-evaluated, and students are given the opportunity to 
conference with peers. 

14. Teacher evaluates paragraphs using same rubric. 

15. Teacher and student meet to discuss evaluation of descriptive paragraphs. 

16. Students set goals for improvement. 

17. Students get a second chance at writing a descriptive paragraph. 

Produced by Eugenie Samara, Durham District School Board, Whitby, Ontario. 
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Exhibit 2 

Evaluation 

Performance/Outcome: Descriptive Paragraph 

Name: Date: 

Rated By: Self Peer Teacher 



1 2 3 4 5 6 

▼ T ¥ ¥ ¥ ¥ 



Criterion 


Low 


Middle 


High 


1. 5 Senses 
Mental Images 

Score: 


minimal use of senses 
or mental images 


some use of the 5 
senses and mental 
images 


excellent use of 
senses and mental 
images 


2. Format/ 
Structure 


little use of format 
or structure 


somewhat/partial use 
of format / structure 


excellent use of 
format / structure 


Score: 








3. Mechanics 
(spelling and 
punctuation) 


quite a few errors 


hardly any errors 


minor or no 
mechanical errors 


Score: 









GOAL: 

SPECIFIC ACTIONS I WILL TAKE... 
1 . 

2 . 

3. 

4. 

5. 



TOTAL SCORE: 



is 



0 



Produced by Eugenie Samara, Durham District School Board, Whitby, Ontario. 
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SELF-EVALUATION RESEARCH PROJECT 

IN-SCHOOL MEETING 



Appendix 1 



Dates agreed upon for meeting(s): __ 

Time: 30 minutes 

The format of each meeting is the same. The process includes 2 parts: 

1) Review by each team member regarding what they did about self-evaluation in the past week and successes 
experienced (e.g. class developed a rubric, 2 short practice sessions using a form from the book, etc.). As each team 
member verbally reports, one team member briefly records the information below. 

2) Each team member brings forth a challenge, problem or concern encountered. The team completes a cycle of barrier- 
busting (see attached sheets), recording the information on the following summary sheet (page 2 of this packet). 

Please bring summary sheets (pages 1 & 2) to our next group meeting. 



IN-SCHOOL TEAM MEETING # 
SCHOOL: 





‘What I did ... 


Success(esl) 


Name: 


► 




Name: 






Name: 






Name: 

0 

ERJC 


is 


18 



Appendix 2 ( a ) 



-Busting (see attached sheet for process directions) 



Name: 


Barrier, Problem or Concern: 




Ideas Generated by Colleagues: 




I think I’ll try ... 


Name: 


Barrier, Problem or Concern: 




Ideas Generated by Colleagues: 




I think I’ll try ... 


Name: 


Barrier, Problem or Concern: 




Ideas Generated by Colleagues: 




I think I’ll try ... 


Name: 


Barrier, Problem or Concern: 

* 




Ideas Generated by Colleagues: 
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